{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More Feature Engineering - Wide and Deep models\n",
    "\n",
    "**Learning Objectives** \n",
    "  * Build a Wide and Deep model using the appropriate Tensorflow feature columns\n",
    "    \n",
    "## Introduction\n",
    "\n",
    "In this notebook we'll use what we learned about feature columns to build a Wide & Deep model. Recall, that the idea behind Wide & Deep models is to join the two methods of learning through memorization and generalization by making a wide linear model and a deep learning model to accommodate both. \n",
    "\n",
    "<img src='../assets/wide_deep.png' width='80%'>\n",
    "<sup>(image: https://ai.googleblog.com/2016/06/wide-deep-learning-better-together-with.html)</sup>\n",
    "\n",
    "The Wide part of the model is associated with the memory element. In this case, we train a linear model with a wide set of crossed features and learn the correlation of this related data with the assigned label. The Deep part of the model is associated with the generalization element where we use embedding vectors for features. The best embeddings are then learned through the training process. While both of these methods can work well alone, Wide & Deep models excel by combining these techniques together. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import shutil\n",
    "print(tf.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load raw data\n",
    "\n",
    "These are the same files created in the `create_datasets.ipynb` notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud storage cp gs://cloud-training-demos/taxifare/small/*.csv .\n",
    "!ls -l *.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train and Evaluate input functions\n",
    "\n",
    "These are the same as before with one additional line of code: a call to `add_engineered_features()` from within the `_parse_row()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CSV_COLUMN_NAMES = [\"fare_amount\",\"dayofweek\",\"hourofday\",\"pickuplon\",\"pickuplat\",\"dropofflon\",\"dropofflat\"]\n",
    "CSV_DEFAULTS = [[0.0],[1],[0],[-74.0],[40.0],[-74.0],[40.7]]\n",
    "\n",
    "def read_dataset(csv_path):\n",
    "    def _parse_row(row):\n",
    "        # Decode the CSV row into list of TF tensors\n",
    "        fields = tf.decode_csv(records = row, record_defaults = CSV_DEFAULTS)\n",
    "\n",
    "        # Pack the result into a dictionary\n",
    "        features = dict(zip(CSV_COLUMN_NAMES, fields))\n",
    "        \n",
    "        # NEW: Add engineered features\n",
    "        features = add_engineered_features(features)\n",
    "        \n",
    "        # Separate the label from the features\n",
    "        label = features.pop(\"fare_amount\") # remove label from features and store\n",
    "\n",
    "        return features, label\n",
    "    \n",
    "    # Create a dataset containing the text lines.\n",
    "    dataset = tf.data.Dataset.list_files(file_pattern = csv_path) # (i.e. data_file_*.csv)\n",
    "    dataset = dataset.flat_map(map_func = lambda filename:tf.data.TextLineDataset(filenames = filename).skip(count = 1))\n",
    "\n",
    "    # Parse each CSV row into correct (features,label) format for Estimator API\n",
    "    dataset = dataset.map(map_func = _parse_row)\n",
    "    \n",
    "    return dataset\n",
    "\n",
    "def train_input_fn(csv_path, batch_size = 128):\n",
    "    #1. Convert CSV into tf.data.Dataset with (features,label) format\n",
    "    dataset = read_dataset(csv_path)\n",
    "      \n",
    "    #2. Shuffle, repeat, and batch the examples.\n",
    "    dataset = dataset.shuffle(buffer_size = 1000).repeat(count = None).batch(batch_size = batch_size)\n",
    "   \n",
    "    return dataset\n",
    "\n",
    "def eval_input_fn(csv_path, batch_size = 128):\n",
    "    #1. Convert CSV into tf.data.Dataset with (features,label) format\n",
    "    dataset = read_dataset(csv_path)\n",
    "\n",
    "    #2.Batch the examples.\n",
    "    dataset = dataset.batch(batch_size = batch_size)\n",
    "   \n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature columns for Wide and Deep model\n",
    "\n",
    "For the Wide columns, we will create feature columns of crossed features. To do this, we'll create a collection of Tensorflow feature columns to pass to the `tf.feature_column.crossed_column` constructor. The Deep columns will consist of numberic columns and any embedding columns we want to create. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 1**\n",
    "\n",
    "In the code cell below you are asked to create some additional crossed feature columns for the model. \n",
    "- For `fc_crossed_dloc` create a crossed column using the bucketized dropoff latitude and dropoff longitude. \n",
    "- For `fc_crossed_ploc` create a crossed column using the pickup latitude and pickup longitude. \n",
    "- For `fc_crossed_pd_pair` create a crossed feature column using the pickup location and the dropoff location. \n",
    "\n",
    "Have a look at [the documentation for Tensorflow's crossed columns](https://www.tensorflow.org/api_docs/python/tf/feature_column/crossed_column) for help."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. One hot encode dayofweek and hourofday\n",
    "fc_dayofweek = tf.feature_column.categorical_column_with_identity(key = \"dayofweek\", num_buckets = 7)\n",
    "fc_hourofday = tf.feature_column.categorical_column_with_identity(key = \"hourofday\", num_buckets = 24)\n",
    "\n",
    "# 2. Bucketize latitudes and longitudes\n",
    "NBUCKETS = 16\n",
    "latbuckets = np.linspace(start = 38.0, stop = 42.0, num = NBUCKETS).tolist()\n",
    "lonbuckets = np.linspace(start = -76.0, stop = -72.0, num = NBUCKETS).tolist()\n",
    "fc_bucketized_plat = tf.feature_column.bucketized_column(source_column = tf.feature_column.numeric_column(key = \"pickuplon\"), boundaries = lonbuckets)\n",
    "fc_bucketized_plon = tf.feature_column.bucketized_column(source_column = tf.feature_column.numeric_column(key = \"pickuplat\"), boundaries = latbuckets)\n",
    "fc_bucketized_dlat = tf.feature_column.bucketized_column(source_column = tf.feature_column.numeric_column(key = \"dropofflon\"), boundaries = lonbuckets)\n",
    "fc_bucketized_dlon = tf.feature_column.bucketized_column(source_column = tf.feature_column.numeric_column(key = \"dropofflat\"), boundaries = latbuckets)\n",
    "\n",
    "# 3. Cross features to get combination of day and hour\n",
    "fc_crossed_day_hr = tf.feature_column.crossed_column(keys = [fc_dayofweek, fc_hourofday], hash_bucket_size = 24 * 7)\n",
    "fc_crossed_dloc = # TODO: Your code goes here\n",
    "fc_crossed_ploc = # TODO: Your code goes here\n",
    "fc_crossed_pd_pair = # TODO: Your code goes here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also add our engineered features that we used previously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def add_engineered_features(features):\n",
    "    features[\"dayofweek\"] = features[\"dayofweek\"] - 1 # subtract one since our days of week are 1-7 instead of 0-6\n",
    "    \n",
    "    features[\"latdiff\"] = features[\"pickuplat\"] - features[\"dropofflat\"] # East/West\n",
    "    features[\"londiff\"] = features[\"pickuplon\"] - features[\"dropofflon\"] # North/South\n",
    "    features[\"euclidean_dist\"] = tf.sqrt(x = features[\"latdiff\"]**2 + features[\"londiff\"]**2)\n",
    "\n",
    "    return features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gather list of feature columns\n",
    "\n",
    "Next we gather the list of wide and deep feature columns we'll pass to our Wide & Deep model in Tensorflow. To do this, we'll create a function `get_wide_deep` which will use our previously bucketized columns to collect crossed feature columns and sparse feature columns for our wide columns, and embedding feature columns and numeric features columns for the deep columns.\n",
    "\n",
    "#### **Exercise 2**\n",
    "\n",
    "In the cell code below has two sets of TODOs fot you to complete. First, you are asked to create two embedding columns. Use the crossed column you created above to create the following embedding columns:\n",
    "- An embedding column of `fc_crossed_pd_pair`\n",
    "- An embedding column of `fc_crossed_day_hr`\n",
    "\n",
    "Look at the documentation of Tensorflow's [tf.feature_column.embedding_column](https://www.tensorflow.org/api_docs/python/tf/feature_column/embedding_column) to make sure you have the correct usage \n",
    "\n",
    "Next, you are asked to add all the necessary numeric feature columns to the `deep_columns` list. Add numeric feature columns for `pickuplat`, `pickuplon`, `dropofflon`, `dropofflat`, `latdiff`, `londiff`, and `euclidean_dist`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_wide_deep():\n",
    "    # Wide columns are sparse, have linear relationship with the output\n",
    "    wide_columns = [\n",
    "        # Feature crosses\n",
    "        fc_crossed_day_hr, fc_crossed_dloc, \n",
    "        fc_crossed_ploc, fc_crossed_pd_pair,\n",
    "        \n",
    "        # Sparse columns\n",
    "        fc_dayofweek, fc_hourofday\n",
    "    ]\n",
    "    \n",
    "    # Continuous columns are deep, have a complex relationship with the output\n",
    "    deep_columns = [\n",
    "        # Embedding_column to \"group\" together ...\n",
    "        tf.feature_column.embedding_column(categorical_column = # TODO: Your code goes here\n",
    "        tf.feature_column.embedding_column(categorical_column = # TODO: Your code goes here\n",
    "\n",
    "        # Numeric columns\n",
    "        # TODO: Your code goes here\n",
    "        # TODO: Your code goes here\n",
    "        # TODO: Your code goes here\n",
    "        # TODO: Your code goes here\n",
    "        # TODO: Your code goes here\n",
    "        # TODO: Your code goes here\n",
    "        # TODO: Your code goes here            \n",
    "        \n",
    "        tf.feature_column.indicator_column(categorical_column = fc_crossed_day_hr),\n",
    "    ]\n",
    "    \n",
    "    return wide_columns, deep_columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Serving Input Receiver function \n",
    "\n",
    "Same as before except the received tensors are wrapped with `add_engineered_features()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def serving_input_receiver_fn():\n",
    "    receiver_tensors = {\n",
    "        'dayofweek' : tf.placeholder(dtype = tf.int32, shape = [None]), # shape is vector to allow batch of requests\n",
    "        'hourofday' : tf.placeholder(dtype = tf.int32, shape = [None]),\n",
    "        'pickuplon' : tf.placeholder(dtype = tf.float32, shape = [None]), \n",
    "        'pickuplat' : tf.placeholder(dtype = tf.float32, shape = [None]),\n",
    "        'dropofflat' : tf.placeholder(dtype = tf.float32, shape = [None]),\n",
    "        'dropofflon' : tf.placeholder(dtype = tf.float32, shape = [None]),\n",
    "    }\n",
    "    \n",
    "    features = add_engineered_features(receiver_tensors) # 'features' is what is passed on to the model\n",
    "    \n",
    "    return tf.estimator.export.ServingInputReceiver(features = features, receiver_tensors = receiver_tensors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train and Evaluate (500 train steps)\n",
    "\n",
    "The same as before, we'll train the model for 500 steps (sidenote: how many epochs do 500 trains steps represent?). Let's see how the engineered features we've added affect the performance. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 3**\n",
    "\n",
    "Have a look at the code below. It should look very similar to the code we developed in the previous notebook for the DNN model we built. The main difference here is that we invoke the `DNNLinearCombinedRegressor` estimator instead of `DNNRegressor` as before. Have a look at [the documentation for the DNNLinearCombinedRegressor](https://www.tensorflow.org/api_docs/python/tf/estimator/DNNLinearCombinedRegressor) and the code for the DNNRegressor we built in the previous lab and fill in the necessary arguments. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "OUTDIR = \"taxi_trained_wd/500\"\n",
    "shutil.rmtree(path = OUTDIR, ignore_errors = True) # start fresh each time\n",
    "tf.summary.FileWriterCache.clear() # ensure filewriter cache is clear for TensorBoard events file\n",
    "tf.logging.set_verbosity(v = tf.logging.INFO) # so loss is printed during training\n",
    "\n",
    "# Collect the wide and deep columns from above\n",
    "wide_columns, deep_columns = get_wide_deep()\n",
    "\n",
    "model = tf.estimator.DNNLinearCombinedRegressor(\n",
    "    # TODO: Your code goes here\n",
    ")\n",
    "\n",
    "# Add custom evaluation metric\n",
    "def my_rmse(labels, predictions):\n",
    "    pred_values = tf.squeeze(input = predictions[\"predictions\"], axis = -1)\n",
    "    return {\"rmse\": tf.metrics.root_mean_squared_error(labels = labels, predictions = pred_values)}\n",
    "\n",
    "model = tf.contrib.estimator.add_metrics(estimator = model, metric_fn = my_rmse) \n",
    "    \n",
    "train_spec = tf.estimator.TrainSpec(\n",
    "    input_fn = lambda: train_input_fn(\"./taxi-train.csv\"),\n",
    "    max_steps = 500)\n",
    "\n",
    "exporter = tf.estimator.FinalExporter(name = \"exporter\", serving_input_receiver_fn = serving_input_receiver_fn) # export SavedModel once at the end of training\n",
    "# Note: alternatively use tf.estimator.BestExporter to export at every checkpoint that has lower loss than the previous checkpoint\n",
    "\n",
    "eval_spec = tf.estimator.EvalSpec(\n",
    "    input_fn = lambda: eval_input_fn(\"./taxi-valid.csv\"),\n",
    "    steps = None,\n",
    "    start_delay_secs = 1, # wait at least N seconds before first evaluation (default 120)\n",
    "    throttle_secs = 1, # wait at least N seconds before each subsequent evaluation (default 600)\n",
    "    exporters = exporter) # export SavedModel once at the end of training\n",
    "\n",
    "tf.estimator.train_and_evaluate(estimator = model, train_spec = train_spec, eval_spec = eval_spec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "Our RMSE for the Wide and Deep model is worse than for the DNN. However, we have only trained for 500 steps and it looks like the model is still learning. Just as before, let's run again, this time for 10x as many steps so we can give a fair comparison."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train and Evaluate (5,000 train steps)\n",
    "\n",
    "Now, just as above, we'll execute a longer trianing job with 5,000 train steps using our engineered features and assess the performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **Exercise 4**\n",
    "\n",
    "Enter the same code you used in Exercise 3 above to train your model for 5,000 steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "OUTDIR = \"taxi_trained_wd/5000\"\n",
    "shutil.rmtree(path = OUTDIR, ignore_errors = True) # start fresh each time\n",
    "tf.summary.FileWriterCache.clear() # ensure filewriter cache is clear for TensorBoard events file\n",
    "tf.logging.set_verbosity(v = tf.logging.INFO) # so loss is printed during training\n",
    "\n",
    "# Collect the wide and deep columns from above\n",
    "wide_columns, deep_columns = get_wide_deep()\n",
    "\n",
    "model = tf.estimator.DNNLinearCombinedRegressor(\n",
    "    # TODO: Your code goes here\n",
    ")\n",
    "\n",
    "# Add custom evaluation metric\n",
    "def my_rmse(labels, predictions):\n",
    "    pred_values = tf.squeeze(input = predictions[\"predictions\"], axis = -1)\n",
    "    return {\"rmse\": tf.metrics.root_mean_squared_error(labels = labels, predictions = pred_values)}\n",
    "\n",
    "model = tf.contrib.estimator.add_metrics(estimator = model, metric_fn = my_rmse) \n",
    "    \n",
    "train_spec = tf.estimator.TrainSpec(\n",
    "    input_fn = lambda: train_input_fn(\"./taxi-train.csv\"),\n",
    "    max_steps = 5000)\n",
    "\n",
    "exporter = tf.estimator.FinalExporter(name = \"exporter\", serving_input_receiver_fn = serving_input_receiver_fn) # export SavedModel once at the end of training\n",
    "# Note: alternatively use tf.estimator.BestExporter to export at every checkpoint that has lower loss than the previous checkpoint\n",
    "\n",
    "eval_spec = tf.estimator.EvalSpec(\n",
    "    input_fn = lambda: eval_input_fn(\"./taxi-valid.csv\"),\n",
    "    steps = None,\n",
    "    start_delay_secs = 1, # wait at least N seconds before first evaluation (default 120)\n",
    "    throttle_secs = 1, # wait at least N seconds before each subsequent evaluation (default 600)\n",
    "    exporters = exporter) # export SavedModel once at the end of training\n",
    "\n",
    "tf.estimator.train_and_evaluate(estimator = model, train_spec = train_spec, eval_spec = eval_spec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "Our RMSE is better but still not as good as the DNN we built. It looks like RMSE may still be reducing, but training is getting slow so we should move to the cloud if we want to train longer.\n",
    "\n",
    "Also we haven't explored our hyperparameters much. Is our neural architecture of two layers with 10 nodes each optimal? \n",
    "\n",
    "In the next notebook we'll explore this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright 2019 Google Inc. Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
